Storage capacities of committee machines with overlapping and non-overlapping receptive fields

We present theoretical investigations via the replica theory of the storage capacities of committee machines with a large numberM of hidden units and spherical weights. Difficulties arise in the solution of this problem in the limit of large M . In the case of overlapping receptive fields, as the number of patterns increases, both permutation symmetry and replica symmetry are broken, which leads to the appearance of many order parameters and causes additional difficulty. We observe that the relations among these order parameters yield a set of quantities which are small in the limit of large M , making the asymptotic calculation tractable. Using the one-step replica symmetry breaking scheme, we compute the asymptotic value αc of the storage capacity per input unit in the limit of large M . We find that αc ' (8 √ 2/(π−2))M√lnM . The shift to the case of non-overlapping receptive fields can be made easily; we then find αc ' (8 √ 2/π) √ lnM . Both values satisfy the bound of Mitchison and Durbin.